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1. Introduction 



The Separation of Variables (SoV) is an approach to quantum integrable systems 
which can be briefly formulated as follows (for a more detailed discussion see the 
survey [H). 

Given a quantum-mechanical system of n degrees of freedom possessing n com- 
muting Hamiltonians 

[H^,H,] = Q, j,fc = l,2,...,n (1.1) 

one tries to find an operator M sending any common eigenvector Px of the Hamil- 
tonians 

HjPx = hjPx (1.2) 
labelled by the quantum numbers A = {Ai, . . . , A„} into the product 

n 

M:Px-^l[Sx-Ayj) (1-3) 

i=i 

of functions Sx-j{yj) of one variable each. The original multi-dimensional eigenvalue 
problem (|1.2|) is transformed respectively into a set of simpler one-dimensional spec- 
tral problems (separated equations) 

"Pj (^Vj, hi,..., hr^j SxjiVj) = (1.4) 

where Vj are usually some differential or finite-difference operators in variable yj 
depending on the spectral parameters hk- In the context of the classical Hamiltonian 
mechanics the above construction corresponds precisely to the standard definition 
of SoV in the Hamilton- Jacobi equation. 

The advent of the Inverse Scattering Method gave new life to SoV providing 
it with the interpretation of the separated coordinates yj (in the classical case) 
as the poles of the Baker-Akhiezer function (properly normalized eigenvector of 
the corresponding Lax matrix). The unsolved question is, however, how to choose 
a correct normalization of B-A function to obtain SoV for a given Lax matrix. 
Nevertheless, as an heuristic recipe, the above idea has proved to be quite efficient 
and allowed to find SoV for a few new classes of classical integrables systems. In 
particular, SoV was found for the systems arising from the r-matrices satisfying the 
classical Yang-Baxter equation in case of An-i (sin) Lie algebra. In the cases n = 2 
and n = 3 the construction of SoV has been successfully transferred to the quantum 
case (see |jl|] and references therein). 

Pursuing the goal to extend the applicability of the B-A function recipe, in our 
previous paper we have studied the A2 Calogero- Sutherland model which does not 
fall into the previously studied cases since it posesses a dynamical (non-numeric) 
r-matrix In the quantum case, our construction of SoV has produced a new 
integral representation for the eigenfunctions of the A2 C-S Hamiltonians (known as 
Jack polynomials) in terms of 3-F2 hypergeometric polynomials. 
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In the present paper we generalize the results of to the 3-particle Ruijse- 
naars model 0] which is a relativistic analog of the C-S model. The corresponding 
eigenfunctions (Macdonald polynomials ^) are g-analogs of Jack polynomials. 
No surprize that the corresponding separated functions are Laurent polynomials ex- 
pressed in terms of 302 basic hypergeometric series. We present also some results 
and conjectures for the general n-particle problem, for instance, we connect the A„_i 
type basic hypergeometric separation polynomials Sx{y) to a terminated case of the 
(pD type g-Lauricella function of n — 1 variables. 

The paper is organized as follows. In Section 2 we describe the classical Rui- 
jsenaars model and, using B-A function technique, construct a SoV. Though the 
results of this section are not used directly in what follows, they provide a useful 
background for subsequent treatment of the quantum case. In Section 3 the standard 
facts concerning the quantum Ruijsenaars model and Macdonald polynomials are 
collected. In Section 4, after introducing the quantum Hamiltonians and Macdonald 
polynomials, we describe the integral operator M performing a SoV and formulate 
the main theorem whose proof takes the rest of the section and part of the next one. 
The main part of the proof is contained in Section 4 where the properties of the 
operator M are studied, whereas in Section 5 the results concerning the separated 
equation (certain 3-rd order g-difference equation and its n-th order generalization), 
as well as its polynomial solutions, are collected. The main technical tool allowing 
us to study the operator M is the famous Askey- Wilson integral identity (|A.15|) . 

Generally, SoV is aimed to simplify the multidimensional spectral problem by 
reducing it to a series of one- dimensional ones. In case of the Calogero-Sutherland 
and Ruijsenaars models, however, the spectrum and eigenfunctions are well known 
and studied by independent means. The main benefit of SoV in application to these 
models is rather producing new relations between special functions. In particular, 
inverting the operator M one obtains a new integral representation for A2 Macdonald 
polynomials in terms of 302 basic hypergeometric functions, which is done in the 
end of Section 4. In section 6 we discuss the obtained results and the possibility 
of their generalization to An, n > 2 case. Two Appendices, A and B, contain, 
respectively, a collection of necessary formulas from g-analysis and some auxiliary 
results concerning operator M. 

During our work we enjoyed the hospitality of the Research Institute for Mathe- 
matical Sciences (Kyoto, Japan) and the Institute of Mathematical Modelling (Lyn- 
gby, Denmark). We express our gratitude to our hosts Prof. T. Miwa and Prof. 
P. L. Christiansen. VBK acknowledges supports from the grant of Forskerakademiet, 
Aarhus, while in the Technical University of Denmark and grant from CRM-ISM as 
a post-doctoral fellow in the University of Montreal. 



2. Classical Ruijsenaars model 

In the spirit of g-analysis, we prefer to use exponentiated canonical coordinates 
and momenta. 
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Definition 1. The variables {Xj, Xj) j = 1, . . . ,n on a 2n- dimensional symplec- 
tic manifold form a Weyl canonical system if they possess the Poisson brackets 

{Xj,Xfe} = = 0, {Xj.Xk] = -iXjXk^jk, j,k = l,...,n (2.1) 

or, equivalently, the symplectic form u is expressed as ui = d\n.Xj A d\n.Xj = 
d{i J2j In Xjd In Xj ) . 

The n-particle (y4„_i) trigonometric Ruijsenaars model [Q is formulated in terms 
of the Weyl canonical system (Tj,tj) where \tj\ = 1, Tj G M (j = 1,2, . . . ,n). The 
Hamiltonians Hj are defined as 



H^= E \ n . ^ = l,...,n, (2.2) 

JC{l,...,n} \ jeJ / / 

\j\=i \fee{i,...,n}\.j / ' 

where ^ ^ 

t'ifc = /_ , , ^ e (1, oo) . (2.3) 

tj tk 

Proposition 1 [|, ^] 

The Hamiltonians Hj Poisson commute. 

{H„Hk} = 0, j,k = l,...,n. (2.4) 
Define the Lax matrix {L operator) by the formula 

L{u) = D{u)E{u) (2.5) 



where 



(f-l)(l-rn) , 



_i + rw + itk 

Proposition 2 

r/ie characteristic polynomial of the matrix L(u) ( |^.5[ ) generates the Hamiltoni- 



ans 



_ rM)(l - n)" det(z - L{u)) 

n 

= ^(-1)'=£V^(1 - iku){l - u)\l - ru^-'^Hn-kZ^ (2.8) 

fc=0 



where we assume Ho = l. 
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In the 3-particle {A2) case which we consider henceforth we have, respectively, 

Hi = t;i2t^l37'l + t^2lf 23T2 + f 31^32^3, (2.9a) 



H2 = V13V23T1T2 + f 12t'32riT3 + f 2lt'3l7'2T3, (2.9b) 

H; = TiT2n, (2.9c) 

D = 2i^{l- U) ^^^S {^12^^13ri,f2lf 237^2, ^31^^32 73} , (2.10) 

i^.. = ^-^, (2.11) 



and 



e{l - uf det(z - L{u)) = z^f{l - - zH\l -u){l- fu)Hi 

+zi{l - iu){l - fu)H2 - (1 - e^ufH^. (2.12) 

To find a SoV for the Ruijsenaars system we use the recipe discussed in the 
Introduction and choose for the separated coordinates Uj the poles upon u of the 
Baker- Akhiezer function il){u) (an eigenvector of L{u)) normalized by the condition 
that its 3-rd component ipziu) is constant. The canonically conjugated (in the Weyl 
sense) variables Yj are chosen as the eigenvalues of L{yj). For the detailed discussion 
of the B-A function recipe see ||l| though the construction described below is quite 
self-contained. 

Define two functions Ai{u) and A2{u) by the formulas 

Ak{u) := Lkk - = Tfc«fc(u), = 1, 2 (2.13) 

-^3,3-fc 

{l-Pn)(thu-h.t){h-eh) 

f(l-.)(Pt3„-t3_,)(ft.-t3)' 

The separated variables Uj are defined from the equation 

Ai{y)=A2{y). (2.15) 
It is easy to see that ( p.l5|) has 3 solutions one of which y = we ignore since 



it is a constant. The remaining two roots we denote yi and y2- From the easily 
verified invariance of ai{u)/a2{u) under the transformation u ^ u '%t2t^^i~^ it 
follows that ^ ^ 

2/12/2 = (2.16) 
The conjugated variables Yj are defined as 

Y,=Ai{y,) = A2{y,), j = l,2. (2.17) 
Equivalently, the four variables Yi,Y2,yi,y2 are defined through four equations 

Y,=Tkak{yj), j,ke {1,2}. (2.18) 
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Theorem 1 The variables Yj, yj satisfy the separated equations 

Yj^ea - y,Y - Y^'fil - - i\)H, 

- - i%)H2 - (1 - fy.YH, = 0, J = 1, 2 (2.19) 

which, by virtue of (^.12^ , imply that det(Y^- — L{yj)) = 0. 



Proof. Substitute into (|2.19|) the expressions (|2.9| ) for Hj and spht the left-hand- 
side of ( |2.19| ) into two terms 

T3Z1 + Y,Z2 = (2.20) 

where 

Zi = -{l-y,){l-fy,)fv3iV32Y^' 

+ (1 - %)(1 - i%)iivuV32TiY, + V2miT2Y^) - (1 - £%)^T,T2, (2.21a) 
Z2 = {1- y.fe'Yf - (1 - y,){l - ey,)e{vi2VizTiY, + t;2it;23T2F,) 

+ (1 - %)(1 - £'i/,)t;i3t;23TiT2. (2.21b) 



To prove ( p.l9| ) it is sufficient to show that Zi = Z2 = 0. Replacing Yj in ( p.21| ) 



by Tiai{yj) or T2a2{yj) in such a way that the factor T1T2 could be cancelled from 
Zi 2 we obtain that Z12 = follows from two algebraic identities for ai 2 

-(1 -?/)(!- £'^y)£'^V3iV32ai{y)a2{y) 

+ (1 - £y)il - fy)iivi2V32a2iy) + V2iVziai{y)) - (1 - f-^y? = 0, (2.22a) 
(1 - yfe^ai{y)a2{y) - (1 - y)(l - fy)f{vi2Vi3a2{y) + V2iV23ai{y)) 
+ il-iy)il-ey)ivi3V23 = 0, (2.22b) 

which are verified directly. ■ 
The third pair of separated variables is defined as 

X := t3, X := T,T2n, (2.23) 

the corresponding separated equation being 

X -H3 = 0. (2.24) 



Theorem 2 T/ie variables {X, Yi,Y2; x, 1/1,1/2) form a Weyl canonical system in the 
sense of the definition 1. 
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Proof. Let us introduce new variables: 



and also 



, _ 1/2 1/2 1 



T+ = T1T2 



1/2 1/2 

y+ = Vi 1/2 , 



1/2-1/2 



1/2 -1/2 



Y- = Y^Y, 



-1 



(2.25a) 
(2.25b) 

(2.26a) 
(2.26b) 



Obviously, (X, T_, T+; x, t_, t+) is also a Weyl canonical system. Note that 

y+ = (2.27) 



because of ( |2.16[ ). Note also that from ( p.l4| ) it follows that Y±, y± depend only on 
T±, t± and do not contain X, x. 

It remains to show that the transformation from {T_,T^;t_,t^) to {Y_,Y^; 
?/_, ?/+) is canonical that is (Y_, F+; ?/_, ?/+) is again a Weyl canonical system. To this 
end, it suffices to construct the generating function F(Y^, y_;t^,tJ) of the canonical 
transformation such that IHl 



i In T-t = t± 



dF 
dtl 



ihiY_ 



-y- 



dF 
dy- 



iliay+ = F4 



dF 
dY 



(2.28) 



and d{F — i\nY+ ln?/+) = z(ln T_(i In t_ + In T+rf In t+) — z (In YLci In + \iaY^dhiy+). 
Recalling the definition of the Euler dilogarithm § 



and introducing the notation 

Ciy; X, y) := Li2{yxy) + Li2(z/xy"^) + lA2{yx^^y) + L\2{vx^^y^^) 

3 — 

we define F := 'ilnyi|_ ln(£~2t^) + F, 

F := z(/:(r^;y_,t_) + £(rV+,t_)-/:(ri;t+,i/_) 

-U2{tl)-U2{t-_^)). 



^2.29) 



(2.30) 



:2.31) 



It is a matter of direct calculation to verify, using (p.l8|) and (|2.27| ), that F 
satisfies ( |2.28| ). ■ 

The identities ( p.l9|) and (|2.24|) and canonicity of the variables (X, Yi, F2; 3;, 
7/2) established above provide, by definition a SoV for the A2 Ruijsenaars system. 
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3. Quantization 



We collect here the standard facts concerning the quantum n-particle Rui- 
jsenaars model 0, |^ and the corresponding Macdonald polynomials p, ||. 

Throughout the paper Z stands for the set of integers, the notations Z>o and 
Z<o are self-evident. 

The quantum Ruijsenaars model is described in terms of the multiplication and 
shift operators, resp. tj and Tj (j = 1, . . . , n) acting on functions of tj 

(t,/)(t) := (T,/)(t) := /(..., gt„ . . .) (3.1) 

(we do not make distinction between variables and operators tj). Here q is the 
quantum deformation parameter related to the Planck constant ^ > as 

q = e-\ ge(0,l). (3.2) 

The operators Tj, tj satisfy the Weyl commutation relations 

[Tj,Tk] = [tj,tk\ = 0, Tjtk = I "j ^ k ^^'^^ 

which produce the Poisson brackets ( p.l|) in the classical limit — > by the standard 
correspondence rule [, ] = —ih{, } + 0{h^). 
The commuting quantum Hamiltonians Hj 

[H,,H,] = 0, j,k = l,...,n (3.4) 

are given by the same formulas ( p.2| ) as in the classical case with the fixed operator 
ordering {Tj to the right). We assume that 

i = q-9 = e9^, g>0, ee{l,oo) (3.5) 

(note that both in the classical and nonrelativistic limits h ^ 0, q = — >■ 1 but in 
the classical limit gf — >■ oo, £ = const whereas in the nonrelativistic limit g = const, 

The operators Hk leave invariant the space Sym(ti, . . . , t„) of symmetric Laurent 
polynomials in variables tj. A basis in Sym(ti, . . . ,tn) is given by the monomial 
symmetric functions labelled by the sequences A = {Ai < A2 < . . . < A„} of 
integers A^ G Z (dominant weights) and expressed as nix = J2ti^ ■ ■ ■ where the 
sum is taken over all distinct permutations v of A. 

Denote |A| = Yl^=i Aj. The dominant order on the dominant weights A is defined 

as 

{n n I 

|A'| = |A|; E^i<E^i> fc = 2,...,n . (3.6) 
j=k j=k ) 

The Macdonald polynomials P^^''^^ g Sym(ti, . . . ,tn) are uniquely defined as joint 
eigenvectors of in Sym(ti, . . . , t„) 

HkPi'-'"^ = hkPi'^'^ (3.7) 
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labelled by the dominant weight A and normalized by the condition 



The corresponding eigenvalues hk are 



hk= E Hi---Hk, fij = q^^f^ ^. (3.9) 
ji<-<jk 

Note that our parameter i and parameter t used in ^ relate as £ = t~^. 
The polynomials P^^''^^ are orthogonal 

1 i i ^pf^)(t)p(f-)(t)A(t) = 0, A^y (3.10) 



(2m 

' \ti\=\ ^ \w\=\ 



with respect to the weight 



see ( |A.2|) for the notation). 



In the limit h 0, g = const the appropriate linear combinations of pro- 
duce the Hamiltonians of the nonrelativistic Calogero-Sutherland model, and the 
Macdonald polynomials go over into the Jack polynomials, see 0. 

In the present paper we consider only the simplest nontrivial case n = 3. 

The Hamiltonians Hk being given by (|2.9|), the formulas ( |3.9| ) produce, respec- 
tively, 

/i^ = + + r /l2 = V"*"^' +?^'"^^' +^"^?^'^^', /i3 = g'^' (3.12) 

for their eigenvalues labelled by the ordered triplets {Ai < A2 < A3} G Z^. 
For instance, 

mooo = 1, "^001 = ^1 + ^2 + ^3, "^011 = tit2 + tits + hts, moo2 = tl + tl + tg, 

mill = tit2t3, "^012 = tit2 + ^1^2 + ^1^3 + tft^ + t2t3 + ^2^3, 
^112 = ^^^2^3 + ^11(2^3 + ^1^2^31 "^022 = ^1^2 ~^ ^1^3 ~^ ^2^3? ^^003 = t^ + ^2 + ^3. 

000 — "^000) -Tool — "^OOl! -foil — "^011) -' 002 — "^002 H ^ ^011; 



p(^;9) _ ™ p(^;9) _ _ , (l-^)(g(2+f)+l+2f) 

-'111 — TTT'lll, -'012 — "^012 H ^i:£2 '^lllj 



-'112 — "^112 5 -^022 — "^022 H ^^112, 

pi^-^l) _ ™ , (l-^)(l+g+g2) (l~£)2(l + g)(l + g+g^) 

-^003 — "^003 H ^23^ "^012 H (g„^)(q2_£) "^111- 
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4. Operator M 



We are now going to describe the integral operator M (|1.3| ) producing the SoV. 
Generally speaking, the kernel M. oi M should depend on 6 variables: M.{x, y 1,1/2 \ 
ti,t2,ts). However, by analogy with the classical case (section 2) and the nonrel- 
ativistic limit [0], it is natural to assume that contains two 5-functions corre- 
sponding to the constraints x = t-^ ( |2.23|) and, respectively, ( |2.16| ). There remains 
thus only one integration in M. Again by analogy with the previously studied cases, 
the kernel is most conveniently described in terms of the variables t± ( p. 25^ ) and 

y± (&)• 

So, let us introduce the operator M 
M ■.^{t,,t2,ts)^^x,yi,y2) 



1 
27ri 



9,29 



■Miiym) 2 , (1/1/1/2)^ I t^)<I/{ih{yiy2)H.,e^2x{yiy2)H_\x) (4.1) 



with the kernel 

M{y+,y- 



{1 - q){q;q)l{tl,t-J;qUC,{i-l;y^,y^i 
2B,{g, 2g) C,{i--2;y^, t^) C,{e-^; t_, y+e 



(4.2) 



where the notation ( |A.7| ) and 



is used. For the definition of the cycle r*|2g 



which depends on g, yi^2 see (|B.4|) and (|A.16| ). 

Remark. In the classical limit, as g — >■ 1, £ = const, using ( |A.2CI| ) and ln£q(i/; x, 
y) ~ —fi^^C(iy; x,y) one obtains that the asymptotics In ~ —ih~^F of the kernel 

is determined by the regular part F (|2.31|) of the generating function of the 
canonical transformation producing classical SoV. As for the nonrelativistic limit, 
h 0, g = const, the easiest way to reproduce the results of [0 is to compare the 
action of the operators M and its nonrelativistic analog on polynomials, see theorem 

i 

Now we are in a position to formulate our main result. 
Theorem 3 The operator M l \4-l\ ) transforms any A2 Macdonald polynomial 



P 



A {ti,t2,ts) into the product 

M : Pi'''^\t„t2,t,) 



c,x^'^Si'-\y,)S^^-\y2 



of functions of one variable only, where the Laurent polynomials S^-^f^x.^ 



A3 



(4.3) 



(4.4) 



fc=Ai 



are expressed in terms of the basic hypergeometric series l \A.di) 



-,1— A21 



q,y 



(4.5) 
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where Xjk = Xj — \k- The coefficients Xx'k' '^i"^ given by 



AA,fc 



-4(P3 



Xi-k 



^1— A31 £2gl— A21 



£2gl-A3i_ 



(?;?)fc-Ai 

The normalization coefficient c\ equals 

MAi - A2 (^'^g)A3l(^~^9)A32(^~^g)A2l 



,1-A21 '9' 9' 



(4.6) 



ca 



(£-3;g),3^(£-i;g),3,(£-2;g),^/ 



(4.7) 



The proof of the above result will occupy the rest of this section and a part of 
the next one. Our proof parallels the similar one for the nonrelativistic Calogero- 
Sutherland model [0]. 

We begin with proving the factorization ( |4.3| ) of MP^''^\ The first step is to 
show that the image MP^^''^^ satisfies certain g-difference equations in x, yi, y2- Let 
us introduce the operators Yj (j = 1, 2) acting on functions of yk as (cf. (|3.1| )) 

(F,/)(y) =/(..., gy„...). (4.8) 

Using y± = {yiyt^Y^"^ (|2.26| a) one can write also 

{Yif){y+,y^) = f{qh+,qh-), {y2f){y+,y-) = f{qh+,q~h-)- (4.9) 

Similarly, 

(ri/)(t+,t_) = fiqh+,qh^), m)it+,t_) = f{qh+,q-h.). (4.10) 

We define also the operator X as X(/)(x) = f{qx). 
Let us introduce the operator expression V 



V{u, z; ni,n2, n^) ■.= (l - qu){l - q'u)tz-^ - (1 - qu){l - qH'u)tz'l-Li 



+ (1 - q(iu){l - q'tu)£zn2 - (1 - qtu){l - q'tu)H. 



(4.11) 



which can be considered as a quantum generalization of the characteristic polyno- 
mial ( 12.121) . The ordering is important in ( [4. 11]) since we are going to replace the 
parameters u, z, TCj by non-commuting operators. 



Proposition 3 The operator M i\4-A ) satisfies the equations 

XM - MHs = 0, 

V{y^, Y.J- MH,, MH2, MH,) =0, j = 1, 2 
where -f^i,2,3 ore the quantum Hamiltonians l \2.!] ). 



(4.12) 
(4.13) 
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Proof. Though the equahty ( [4.12[ ) is easy to derive from the fact that M respects 
the constraint x = t^, or directly from ( [4.1|) , we shall proceed, however, in a more 
methodical fashion allowing to prove both ( [4.12|) and ( [4.13| ) in the same way. Let 
us rewrite first the operator identities ( [4.12| ) and ( |4. 13| ) for M as algebraic identities 
for the kernel M (^) . 

We define the Lagrange adjoint Hamiltonians as 



Hl=T^ ^Vi2Vi3 + T2 ^V2lV2'i + T3 ^f3lf32, 



(4.14a) 
(4.14b) 



H;=T{^T^^T^\ (4.14c) 



(4.15) 



In particular, T* = T- ^. Considering M in (|4.12| ) and ([4.13|) as integral operator 



we can use integration by parts and switch to the kernel M. replacing them by 
H^. according to ( 4.15| ) which results in the g-difference equations for M.: 



0, 



1,2. 



(4.16) 
(4.17) 



(X - Hl)M 

While ( |4.16| ) is obvious, ( |4.17| ) needs more consideration. Note that, by virtue of 
and ( [4.10| ), the action of V on M.{ii^,y_ \ t^) is well defined. Note also that 
the equations ([4.16|) and ([4.17|) are the quantum counterparts, resp., of the classical 
separated equations (|2.24| ) and ( p.l9|) . 

The next step is to notice that the kernel ( [4.2|) satisfies the four first order 
g-difference equations 



Y,TuM = ak{yj)M, j,A;G{1,2}, 
where (compare to classical ( p.l4|) ) 



akiy) 



£{1 - y)iqitk - ts){qi%y - h^k){tk - h-k) ' 



k = 1,2 



(4.18) 



(4.19) 



which are verified directly from ( [4.2| ) using the relations (|A.4| ). Note that ( |4.18| ) is 
the quantum counterpart of ( p.l7| )- (|2.18| ). 

Remark. It is easy to verify that the system ( ^4.18| ) is holonomic, that is the 
operators akiij j)~^YjTk commute, provided yj and are bound by ( p.l6|) . 

We proceed now to derive the third-order g-difference relations in yj ( 4.17| ) for 
M. from the first-order relations ( [4.18|) . The proof parallels that of theorem || for 
the classical case. Let us write down the equations ( [4.17| ) explicitely 

;i - qy,){l - q\)eYf - (1 - qy,){l - qH\)fYfHl 

+ (1 - g%)(l - qH%yY^H; - (1 - qey,){l - qH^yj)Hl\ ^ = 0, (4.20) 
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then substitute into ( |4.2(J| ) the expressions ( |4.14] ) for H* and spht the left-hand-side 
of ( [4. 201) into two terms 



T^^Zi + YjZ2 = (4.21) 
where 



Z, = [-{1 - qyj){l - q'i'y,)i'YfvsiVs2 - (1 - ql%)il - qH%)T^^T^' 

+ (1 - q(iy,){l - qH%)iY,{Tr'vuVs2 + T2'v2iV3i)] -M, (4.22a) 

Z2 = [(1 - %)(1 - qyj)i%' + (1 - %)(1 - qfyjm-%-'vnV23 

-(1 - y,)il - qfy,)£%iT-'vuVi3 + T-\2iV23)] M. (4.22b) 
Introducing the notation 

ai2{y) = ai{qy\.^^^^a2{y) = a2{qy\,=qt^ai{y) , (4.23) 

1 1 
i ^tj — qi^tk 



(4.24) 



and noting that 

TkVjk = VjkTk, VjkTj = TjVjk, (4-25) 
it is easy to verify the algebraic identities for ai^2 

-(1 - qy){l - qH'y)fvsm2ai2iy) - (1 - qfy){l - q'i'v) 

+ (1 - qiy){l - q^fy)i{vi2V32a2{y) + «2i^3iai(y)) = 0, (4.26a) 

(1 -y){i- qy)i'ai2{y) + (1 - iy){i - qi'y)iv,,v23 

-(1 - y)il - qfy)fivi2Vi3a2iy) + V2iV23aiiy)) = 0. (4.26b) 

Now we insert TkTj^^ in appropriate places in ( [4.22| ) in such a way that Ti^T2^ 
could be carried out to the left of [...]. Then we push the products YT to the 
right using ( |4.25| ) until they hit A4, so that ( [4. 181 ) could be applied. The equalities 



Zi = 0, Z2 = and therefore ([4.201) and ( [4.13|) follow then immediately from (|4.26|) . 
■ 

Proposition 4 The function {MP^''^^){x,yi,y2) satisfies the q-difference equations 
(separated equations) 

{X - h^)MPf''^'^ = 0, (4.27) 
V{y„ Y,- hi, h2, hs)MPi'-''^ = 0, j = 1, 2. (4.28) 
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Proof. Apply the operator expressions ( ^.121 ) and ( [4.13[ ) to the function P; 



Using the operator ordering convention and the fact that Macdonald polynomials 
P^''^^ are the eigenfunctions of the Hamiltonians Hj ( |3.7|) one replaces Hj by hj. 
Since hj are just numbers, the operator M can be applied then directly to P^^'"^^ 
which results in ( |4.27|) and (|4.28|) . 



In order to derive the factorization ([4.3|) of MPf''^^ we need more specific in- 
formation about how M acts on the symmetric polynomials from Sym(ti, ^2, ^s)- 
Note that solutions to (|4.18| ), as to any g-difference equations, are defined only up 



to a factor invariant under g-shifts (quasiconstant). Our choice ( ^.21 ) of the kernel 
M. corresponds to a particular choice of the quasiconstant which is crucial for the 
results given below. 

Since the kernel 7V1 ( [4 .21 ) is a particular case ( |B.8| ) of the kernel A^a/j ( |B.7| ), we 
can make use of the results obtained for M.ai3 in Appendix B. 

Let us define few polynomial spaces. Let Sym(ti, ^2, ^3) be the space of Lau- 
rent polynomials symmetric w.r.t. permutations of 3 variables ti, ^2, ^3- A basis 
in Sym(ti, ^2, ^3) is given by mx or P^'''\ Let Sym(ti, ^2; ^3) be the space of Lau- 
rent polynomials of the same 3 variables, symmetric only w.r.t. ti t2- Obvi- 
ously, Sym(ti, ^2; ^3) 3 Sym(ti, ^2, ^s)- Though the Macdonald polynomials belong 
to Sym(ti, ^2, ^s) it is convenient to define M on a larger space Sym(ti, ^2; ^3)- 

Let Ref t+; ts) be the space of Laurent polynomials in t±, which are refiexive 
in t- (invariant w.r.t. t_ tZ^) and even in t± (invariant w.r.t. (t_,t+) —>■ (— t-, 
— Note that the change of variables (ti, ^2, ^3) — > (t_, t+, ta), see ( ^.25| a), provides 
an isomorphism Sym(ti, ^2; ^3) — Ref (t_; t+; ^3). 

The spaces Sym(j/i, 2/2; x) ~ Ref (?/_;?/+; x) are defined similarly. 

Proposition 5 

M : Sym(ti,t2;t3) Sym(yi, 1/2; a;). 

In particular, the image of a Macdonald polynomial p^^''^^ g Sym(ti, ^2, i^s) (^Iso 
lies in Sym(?/i, 7/2; a;). 

Proof. The proposition |T3| from Appendix B implies that M : Ref (/!:_; t+; t^) — > 
Ref (?/_;?/+; x). Using the isomorphisms Sym(ti, ^2; ^3) — Ref (t-; t+; ^3) and Sym(?/i, 
1/2] x) ~ Ref (?/_; y+; x) we conclude the proof. ■ 

Now everything is ready to prove the main statement of the theorem |^. 

Proposition 6 The operator M transforms any Macdonald polynomial P^^'"^^ into 
the product U-^ )- 

Proof. We have already established that MP^^''^^ is a Laurent polynomial (proposi- 
tion |]) satisfying the g-difference equations ([4.27|) and (|4.28| ). The factorization 



follows from the fact that x^^ and S^''^\y) are the unique, up to a constant factor, 
Laurent-polynomial solutions, of the g-difference equations, resp. {X — h'i)f{x) = 0, 
and V{y,Y;hi,h2,h3)f{y) = 0. The first statement is obvious, as for the second 
one, see proposition |12l ■ 
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Though for the theorem^ we have used only the polynomiahty of MP^^'''\ in fact, 
the action of M on Sym(ti, ^3) can be described in much more detail Namely, 
taking the formula ( [B.13|) from Appendix B, making the substitutions (p3.8| ) and 
performing the changes of variables (|2.25| a) and (|2.26| a) one obtains the following 
result. 

Theorem 4 Consider the basis in Sym(ti, i^a) 

Pjku:=ti-^''t'^4{r\t^\e^H2ts^;q)u, 3,keZ, u e Z>o, (4.29) 
and in Sym(?/i, 1/2; x) 

Pjku ■■= x^y'ly^ivi, 2/2; q)u, j,k eZ, u E Z>o, (4.30) 
respectively. The operator M acts on pjkv as follows 

M : p,.. ^ ^''l^f^.^- (4-31) 



Postponing the proof of the formulas ( [4.5|) and (|4.6| ) for the next section, we can 
prove now the final statement of theorem ^. 



Proposition 7 The normalization coefficient c\ in ( l^.^j ) is given by ( [j-Tj ) 



Proof. In this case, it is convenient to make use of the isomorphisms described above 
and to think of M as acting from Ref (t_; t+; ^3) into Ref (?/„; x). Comparing the 
asymptotics of the monomial symmetric functions mx 

and of the polynomial pjku (|4.29| ) 



p^,, = titf{i-%t.,rH+tz')u 

~ {-iYq'^r''titf+''t''^, t_ ^ 00 

we conclude that the transition matrix between the bases mx and pjku is triangular 

mx = (-i)^-g''''^^^-P|Ai,A„A3. + E E(- • (4-32) 

iy<X3i j,k 

Given the mutual triangularity ( |3.8| ) of the bases P^^''^^ and mx, it means that 
the expansion of P^^'^-' in pjj^^ has the same structure as ( |4.32| ). Using then ( [4.31| ) 
and the asymptotics of pjku ( [4.30|) 



Pjku = x^y+{y+y-, y+yZ\ ~ (-l)"g'^x^y+ y^ ^ 00 
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we obtain 



,?)a31 

On the other hand, ( [4 .41) imphes that 
whence 

CA xf£ ^ = f's't^"'^ ^^-^^^- • (4.34) 

It remains only to use the formulas ( [5.14| ) proved in the next section, and obtain 

&■ ■ 

Compared to our formula i\i.7\} for the normalization coefficients ca is new, 

and its nonrelativistic analog 

(2fi')A3l(2fi')A32(fi')A21 / X _ / , T\ / , , T\ 

(3^)a3i(^)a32(2^)a2i 

fills the gap in the description given in of the integral representation for Jack 
polynomials analogous to ( [4.38|) . 

We conclude this section with a list of results concerning the inverse operator 
M~^. All the preparatory work being done in Appendix B, it remains only to use 
the correspondence ( |B.^ ) between M^/? and M. 

From ( p.20|) and ( p.21|) it follows that is an integral operator 

Af-':t(l,t,i,t,2)-*(«i,(2,«3) 

1 r rfy-.^fMii.f^iV , Lf(,,!ZM)!lh,!ZM)i| (4.35) 



g) ^ (^_^)A3i^-^Si(M^^2Ai+A3(lAil^ 



r*+'l- 
-9,39 

with the kernel 

m., t- 1 V-) = (^-<i)(<i-^<iy^(y-;y~''^ (4.36) 

Reversing (^4.31|) one obtains the formula for the action of on the basis pjku 

Reversing ( [4 .31 ) provides a new integral representation of A2 Macdonald polyno- 
mials in terms of the Laurent polynomials Sf''^\y) ( [4.5| ) 

: c^x\^\sf'^\y^)Sf''\y,) P^^''^\t,,t,M). (4.38) 
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Finally, from the propositions O and |T3| it follows that for positive integer g the 
operator turns into a g-difference operator of order g: 




(4.39) 



where ^A;(r, s) is given by ( [B.15|) . The result is not surprising in view of the similar 
result for the nonrelativistic case [Q] where becomes a differential operator of 
order g for g E Z>o. In this result was derived using a representation of 
in terms of the fractional differentiation operator. In the relativistic case it is also 
possible to relate with a sort of fractional g-difference operator. We intend to 
touch this subject in a separate paper. 



5. Separated equation 



In this section the results are collected concerning the Laurent polynomials 
^x'^^y) ^'^d the corresponding g-difference equations. Since all the results are easy 
to generalize from n = 3 to arbitrary n, we give them in the most general form. 

Conjecture 1 The correct generalization of the formula for S^^'''\y) for any 
n is given by 



Sx'''\y) = y^'iV, q)l-ngn<Pn~ 



where 



fll, . . . , Ojji 
bi, ■ ■ ■ , bn-l 



bj = aj 



q,y 



(5.1) 
(5.2) 



Proposition 8 Sx'''\y) is a Laurent polynomial in y of the form 

si'''\y) = E xffy'- (5.3) 

fc=Ai 

Proof. Observe, first, that if a = bq" for some positive integer u then 

{a;q)k {bq'';q)^ 



{b;q)k {b;q)^ 



(5.4) 



is a polynomial in q^ of degree u whose coefficients are rational functions in b and 
q. As a consequence, if aj+i = bjq^^ then 

p^qk^ _ {a2]q)k---{an]q)k ^ (6ig^ g)^, . . . (fe^-ig^ g)^„_i g^ 

^ ~ {h]q)k- ■ ■{bn-i;q)k {bi;q)ui- ■ ■{K-i;q)u„^i 
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is a polynomial in of degree N = vi + ■ ■ ■ + Vn-i- 

In our case, Vj = Xn-j+i — A„_j, = A„ — Ai by virtue of {f).2\j, and from ( |5.1| ) 
and (|A.9|) one obtains 



n(Pn-l 



ai, 



q,y 



fe=0 W,9Jfc 



• • • ! &n-l 

where P^iq^) is given by (|5.5|) . It remains now to apply the following lemma. 
Lemma 1 Let PN{y) be a polynomial in y of degree < N. Then 



(5.6) 



where QN{y) is a polynomial in y of degree < N. 



(5.7) 



Proof. It is sufficient to consider the polynomials PAr(g^) = [q^ for 
u = 0,1, . . . , N forming a basis in the polynomial ring. Then 



^ y\q'-^'; g). = E [■■■] = E 4^ y' 



y 



k=v 



fc=0 



(?; ?)fc-!. 



A:=0 



(?;?)a 



(5. 



Using the formula ( |A.11|) and the identity {aq"; q)oo = [o-'f] <i) n -ui^CLI^ ] <l)oo one 
obtains finally the expression ( |5.7| ) where QN{y) = («-; q)uy'^iciq'^y', <i)n-v ■ 

Applying the above lemma to the case of the polynomial P^lq^) given by (|5.6| ) 
and a = ai = £^q^^-^n+i obtain finally that y~^^ S^''^\y) is a polynomial of 
degree < A„ — Ai. ■ 



Proposition 9 The coefficients Xa the expansion ( \5. ^ ) are given by 



XA,fc - J 7— N n+l(pn 

[Q, QJk-Xi 

In particular, for n = 3, (j3T^ produces (\4-(\ ) 



-,Xi-k 



, Cll , . . . , ujyi 



(5.9) 



Proof. We know already that S^'''\y) is a Laurent polynomial and is thus defined 
for any y G C \ {0,oo}. Suppose for a while that \y\ < 1. Then both factors 
{y;q)i-ng and n(pn-i in ( p.l| ) are given by the convergent series ( [A.llD and ( |A.9| ), 
respectively. Multiplying the two power series in y we observe that the coefficients 
at y'' is expressed in terms of n+i0n series: 



si'-''\y) = y'^T. 



k=0 



,n.k {q-'^-'';q)k 

iQ;(i)k 



q , Oi, . . . , a„ 

'''+H^,bi,...,bn-l 



(5.10) 
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In fact, the sum in ( [5.10D is finite: Ylk=o- '^^ ^^i^' formula (1.9.11) 

from [0: let u, ki, kn & Z>o, then 



n+10n 



Ol , . . . , On 







(5.11) 



ioT u > ki + ■ ■ ■ + kn- Substituting 



V = k 



bn = q 



,2—k nn 



kn = \\-\n^k-\ 



i+1 



A 



■n-j, 



we obtain that 



— fe 

g , ai, . . . , a„ 



j = 1, . . . - 1 







for fc>A„ — Ai + 1, hence the sum in ( |5.1CI| ) is finite: I]fc=o- The coefficient at in 
(15.101 ) produces, respectively, ( ^.9|) . ■ 
For the sake of reference we present a short list of polynomials S^^''^\y) in case 
n = 3: 

^ 1 



'-'000 



^002 



1 + 5£f^ = l + £(£+l)2/, 



{P-q)(l+l) 



'-'012 — -Li JT^ r ^ y ) 



'-'022 



C(^;9) _ 1 I (l+g+g^)(i;-l)^^y , (l+q+g2)(^-l)^4^2 (<?-g2)<;6i;3 
'^003 — J- ^2_g2 (£+^)(£2_g) (^+l)(^+q)(^2_g) , 

cC^;-?) _ 1 , (£3+g^^2+£^g+^2-g^^-^g-^-g^)£y 
•JOIS — -Li £23^2 

I (g£3+£3^£2g2^£2g_)_^2_^3^_^g2_^g_^3_g2-)(^_]^)^3^2 (^-g)£5^3 

C<(^;g) _ 1 I (g^3^£3_|_£2^2_|_£2g_|.£2_g3£_g2£_£g„g3_g2-)(-£_]^)£^ 
'^023 - -L + (£_<^)2(£+g) 

I {fi+Pq'^+Pq+l'^-q'^i-iq-l-q'^){P-qY'^y'^ , {f-q)l-^y'^ 



(£_g)2(^+,) 



£-g 



5, 



033 



^-g2 (f-g){^-g2) 



(<?-+!) (£+g) (£2 -g)f 



Remark. It easy to give more simple expressions for some of x^\k ^^^h as 



AA,Ai — -L) 



and 



n-l 



7=1 ^';g)A,+i-Ai(^ ■'';?)a„-a„. 



(5.12) 
(5.13) 
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which for n = 3 produce 



-2A1+A2+A3 ;?)a2i 



(5.14) 



The formula ( |5.12| ) is obvious. To obtain ( [5.13| ), use the summation formula 
(1.9.10) from ^ 



{q; q)u{l3i/l3; q)k^ . . . (/3„-i//3; g)fc„_i 



(5.15) 



{l3q; q)y{l3i] q)k, ■ ■ ■ (Pn-i, q)k„^, 
where i/, fci, . . . , kn-i G Z>o and u > ki + ■ ■ ■ + fc„_i. Substituting 

u = A„i, (3 = Tgi-^- = ai , 
= £J^i-A.+i+Ai ^ ^ ^^.^^ _ ^^.^ J = 1, . . . , n - 1 , 

we obtain, after a series of equivalent transformations (see Appendix I to ^^), the 



expression ( |5.13| ). 

Remark. There is also a simple formula for Sx''^\^~"')'- 



5f^)(r") = r"^nr-g)A„,n 



(r^;g)A,-A, 

j=\ {^~^-,q)x,+i-xi 



(5.16) 



or, for n = 3, 



ii-';q)xA^-';q)x,,' 

which are proved in a way similar to (|5.13|) using the formula (1.9.9) from flO 



(5.17) 



The polynomials S^^''^^ (y) can be expressed also in terms of the g-Lauricella func- 
tion (WJ^. 



Proposition 10 The following equalities hold: 



Ai 



np(g'-""-+^+^^"-^; g)A„_,^.-A„_, 

n'- h' h' 



where Xij = Aj — A-,- and 



q, Xi , . . . , Xfi—x 



a =y, c = qt q ^"y, Xj = qt ■'q , _ ^ 



n—j^Xx—Xn-j y _ ^Xn-j—Xn-j + l 

mi 



(5.1^ 



(5.19) 



for j = l,...,n — 1. Another expression for ' (y) reads 



Sx'''^\y) = y^'iq^y,q)ii-n)g ( Y[iauq)g ] <Pd 



/n— 1 



qly 



g; ai, . . . , a„_i 



(5.20) 
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Proof. The formula is obtained by substituting the parameters ( |5.19| ) into 

Andrews' formula ( |A.13| ) for g-Lauricella function and comparing the result to (|5.1|) . 

Note that c/a' = ai, Xj = ctj+i, b'jXj = bj. 



Similarly, substituting into ( |A.13| ) the parameters a' = y, c = qiy, b'j = 
= Gj (j = 1, . . . , n — 1) such that c/a' = an, b'-xj = bj, one arrives at (|5.2CI|) . 



Corollary 1. Substituting the definition ( |A.12| ) of (pD into formula ( |5.18| ) we 
obtain another explicit representation for S^''^\y): 



n—j-\-\ ^n — j 



■^72 — An— 1 

fci=0 



A2— Ai 

E 

fcn-l=0 



(7ngAi„+fci+...+fc„ 



^U'-, Q')A„i-fci-...-fc„_i q')fci + ...+A:„_i 
'"-1 (g^n-j-A„_j+i. q-^^^q£n-jqXi-\„-j-^kj 



(5.21) 



Corollary 2. It is possible also to represent Sx''^\y) as a g-integral ( |A.8| ): 



1 



X 



Ai„ + 1 - ng) Jo 



"9'' T-jn-l 



1 \IJ Xn — j + l~^n — j 

(^9; 9)Ai„-ng 



n"=i(tg^"-^g^i-^"-^;g)A„. 



(5.22) 



To obtain the formula ( p.22| ) rewrite Andrews' formula ( [A.13| ) as a g-integral 



, q, X\ , . . . , 



1 



Bqia^'y - a) Jo 



1 ^ 1-rn-l 



7— o— 1 



and substitute 

a = x {y = q''), = Xin + l-ng + x, Pj = Xn-j- Xn~j+i, xj 



np'(te,;g)^, ■ 
(5.23) 



The rest of the results are concerned with the separated g-difference equations 
for the polynomials S^''^\y). 

Proposition 11 The polynomial f{y) := S^^''^\y) ( \5. i|j satisfies the q- difference 
equation 



E(-l)'^'^'(l - q'(^'y){y; q)k {q^^'ty- q)n-k h^-k f{q'y) = 



(5.24) 



A:=0 



where, are given by i\3. f\ ) and, as in the classical case (2^), we assume Hq = 1. 
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Proof. Using the definitions ( |5.1| ) and ( p. 21 ) together with ( |3.9D , it is a matter of 
straightforward calculation to transform the g-difference equation ( |A.1CI|) for n4>n-i 
into m 

In fact, the factor (1 — q^t^y) can be cancelled from ( |5.24] ) which results in the 
equation 



+ E(-l)'^^'(l - /^^)(^/; <l)k [q'^'ty- q)n-k-i K-k f{q'y) = . (5.25) 

k=0 

In the case n = 3 the g-difference equation (|5.24|) takes the form 

V{y,Y; h,,h,h,)f{y) = 

where V is given by ( |4.11|) , or, explicitely, 

(1 - qy){l - q'y)f f{q'y) - (1 - qy){l - q'fy)fh fiq'v) 
+ (1 - qiy){l - q'fy)ih2f{qy) - (1 - qfy){l ' q^(-h)hzf{y) = 0. (5.26) 



Proposition 12 Let 

Gr:=ZuizU...U^Z, G«:={1,^,...,^|. 

2 n — \ [n n n ) 

Then, for all g > except for the finite number of points g E Gn = G!^^ fl G^\ 
the separated polynomial f{y) := S^)^''^\y) l \5.1\ ) is the only, up to a constant factor, 
Laurent-polynomial solution to the q- difference equation ( \5.24) - 

In particular, G3 = 0, so for n = 3 the uniqueness of L.-p. solution holds ^g > 0. 



Proof. In the nonrelativistic case the analog of the equation ( |5.24| ) is a differential 
equation having 3 regular singularities: 0, 1, 00, and the uniqueness of L.-p. solution 
is proved by analysis of the corresponding characteristic exponents. As shown below, 
the argument can be translated rather directly to the g-difference case. 

Let f{y) be a non-zero Laurent-polynomial solution to ( p.24| ) or, equivalently, 
(|5.25| ). Then, subsituting into (|5.25|) the values y = q^^ , j = 0, 1, 2, . . ., one observes 
that f{q~^) can be determined recursively, starting from /(I) since the factor (|/; q)k 
cuts away the terms with k > j. The only obstacle could be the vanishing of the 
factor {q^^^P^y]q)n-k-i for = which may happen only for g G G^^\ Suppose 
g ^ G^^\ Then it is sufficient to use the fact that any Laurent polynomial vanishing 
on a countable set vanishes identically. It follows that, first, /(I) 7^ for any non- 
zero L.-p. solution and, second, any two non-zero L.-p. solutions are proportional, 
in particular to the standard solution sf''^\y). 

Instead of the sequence y = q~^ one can take y = qH~^ and use the same 
argument. Note that the above recursive process is the exact analog of the Taylor 
series expansion around y = 1 'm the nonrelativistic case. 
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On the other hand, a similar argument works with expansion around or oo. 
Substituting in (|5.24|) the expansion f(y) = Y.k=k. fkU^ obtains (n + 2)-terms 



recurrence relation YIj=l Akjfk_j = for f^. The "boundary" coefficients A^o and 
Ak,n+i have simple form 

n 

Ako = i-iri"^ n {q" - q^^e-^), (5.27a) 

n(n + l) 

Au,n+^ = - j n(g' - g^^+"+^r-^). (5.27b) 

Suppose g ^ C^^^ Then, since £ = q~^, the coefficient A^o vanishes only for 
k = Ai, and A^^n+i = only for = A„ + n + 1. Hence inevitably A;_ = Ai, k+ = A„, 
and the coefficients fk are determined recursively in a unique way starting from /a^ 
or /a„ which proves the uniqueness of L.-p. solution. ■ 

The question whether the uniqueness of the L.-p. solution really breaks for g G 
Gn, remains still open. 

It would be interesting to strengthen the above result. 



Conjecture 2 The equation ( \5. 2^ ) with free parameters hj has a polynomial solu- 



tion only for hj given by ( \3. dj ) and A = {Ai < A2 < . . . < A.„} G Z". 



6. Discussion 



The results of the present paper generalize to the case of the Ruijsenaars model and 
Macdonald polynomials those of P| obtained for the Calogero-Sutherland model and 
Jack polynomials. In the nonrelativistic limit h ^ 0, g = const, the Hamiltonians 
Hk, operator M, separated polynomials S and equations for them go over into the 
corresponding objects described in 

The crucial element of our approach is the operator identity ( [4.13| ) which al- 
lows to prove the factorization (^4.3|) of MP^^''^^ and to establish thus the separation 
of variables. The identity ( [4.1 3| ) is apparently a quantum analog of the charac- 
teristic equation for the classical Lax operator. Moreover, the kernel A4 can be 
considered as a collection of eigenf unctions to the quantized separation variables 
yj describing thus the change of basis from 't- representation' to 'y- representation'. 
Though these analogies with the classical inverse scattering method proved to be 
useful as an heuristic tool for finding SoV for quantum integrable systems [||], their 
algebraic/geometric origin is still to be cleared up. 

An interesting problem is to search for alternative forms of M. We have presented 
here two descriptions of M: analytical ( [4.1D in terms of Askey- Wilson integral, and 
algebraic ( [4.31| ) in terms of the basis pjku- Our study of M is based mainly on the 
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analytical definition. It would be interesting also to develop the theory of M based 
entirely on the algebraic definition, in particular, to give a purely algebraic proof of 
the identity ( ^131) . 



When our work was close to be finished we became aware of the preprint |]T^ 
of Mangazeev addressing the same problem of SoV for A2 Macdonald polynomials. 
His proposal for the operator M is different from ours, using a g-integral rather than 
a contour integral as we do. Some of his arguments are quite formal, for instance, 
expressions with the gV'e-series he is using as a final result are divergent. It seems 
that our choice of M, compared to that of fl^, allows to overcome the problems of 



convergence of the g-integral and to obtain explicit expressions for and action 
of M on polynomials. Still, the problem of representing M as a g-integral seems to 
deserve a further consideration. 

Although we can predict the form of the separation polynomial S^l'''\y) for 
the n-particle case and study it in detail (section 5), the corresponding n-particle 
generalization of the kernel M. is not yet clear, so it is an open problem to separate 
variables for the An-i Macdonald polynomials for n > 3. 

In fact, there are infinitely many "separating" operators M^'^\ since for any 
choice of c\ the operator defined as 



n-l 



M('^) : Pf^'\t,, . . . - c,x\'\ n Sf^'^\y,) (6.1) 

will serve the purpose. The genuine problem, however, is to choose the coefficients 
c\ in such a way that the corresponding kernel A^*^") were given by an explicit 
expression generalizing 



Appendix A. Formulas from analysis 

For reader's convenience, we have collected here the most important definitions and 
formulas from g- analysis used in the main text. For references see |p!0| , |Tl|, [1^, 
Especially useful for practical calculations is the collection of formulas in Appendices 



I and II from |T0|. Throughout the text it is assumed that < g < 1. 



The g-shifted factorial and its generalizations are defined as 
(a;g)o:=l, (a; g)^ := (1 - a)(l - ag) . . . (1 - ag'^"^). A; = 1, 2, . . . , (A.l) 

(a;g)oo:=n(l-«9'), (x; g). = ^ T > « ^ C, (A.2) 

fc=0 W -^1 Q'Joo 

(ai, 02, ■ ■ ■ , a„; q)k ■= (cn; g)fc(a2; q)k--- (an, q)k, /c = 0, 1, 2, . . . or 00. (A.3) 
Note the useful relations 

{qx; q)a = ^— ^(x; g)„, (g^^x; g)« = ^_ (x; q)^. (A.4) 
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We make use also of the g-binomial coefficient 



g-Gamma and g-Beta functions 

iq-,q)ooil-qy-'' 



/c = 0, 1, . . . , n, 



r,(^ + i) = V^r 

1 — 



(A.5) 



R(n M r,(a)r,(&) (g,g'^+^g)oo 



r,(a + 6) 



(g»,g^g)c 



g-integral 



„1 CX) 

^0 ^0 



(A.6) 
(A.7) 

(A.8) 



and basic hypergeometric series {n G Z>o 
O]^ , . . . , Ci„ 



6l, . . . ,6„_ 



(cf-i, ■ ■ ■ 5 ct^; g)fc 



k=0 



(g,6i,...,6„_i;g)fc' 



< 1- (A.9) 



Denoting the expression ( A.9 ) by f{y) we observe that it satisfies the n-th order 
g-difference equation, see and [0 (section 2.12.3): 



y n (1 - ^kY) - n (1 - <l''bkY) fiy) = 



(A.IO) 



k=l 



k=l 



where {Y f){y) := /(gy) and 6„ = g. 

Summation formula for i^o (g-binomial series): 



i<Po 



A;=0 



iq;q)k iy;q)c 



\y\ < 1- 



(A.ll) 



The (/)D-type g-Lauricella function [[l3i [15| of ra — 1 variables IS cl mult i- variable 
generalization of the basic hypergeometric series: 



■>D 



a ]b^, . . . , 



g, Xi , . . . , Xfi—i 



E 



(a'; g)fci 



n-1 fU 



{b'j;q)k^x/ 



fci,..,fc„_i=o (c; '?)fci+...+fc„-i j=i ('?;9)fcj 

(A.12) 

Andrews has found that (pD can be expressed in terms of the basic hyperge- 
ometric function n4>n-i of one variable: 



a ;bi, . . . , 



X 



, g, Xi , . . . , Xfi—i 

c/a', Xi, 

b[xi, . . 



{a',b[xi,...,b'^_^Xn-i]q)c 

(c, Xi, • • • ) Xyi~\i g)oo 

•^n— 1 / 
'-'n-l-^n-l 



(A.13) 
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Our main technical tool, on which the proof of the main theorem ^ depends, is 
the famous Askey-Wilson integral (|jlO[, section 6.1; [jlT|, section 2.6). Let 



w{a, b, c, d; t) :- 



Then 



(at, at ^, ct, ct~^, dt, dt~^; q)oo 

1 f dt 2{abcd] q)^ 

— J jw{a,b,c,d;t) 



(g, ab, ac, ad, be, bd, cd; q)c 



(A.14) 



(A.15) 



The cycle Tabcd depends on parameters a, b, c, d and is defined as follows. Let 
Cz,r be the counter-clockwise oriented circle with the center z and radius r. 

If \a\ , \b\ , \c\ ,\d\ < 1 then Tabcd = Co,i- The identity (|A.15|) can be continued 
analytically for the values of parameters a, b, c, d outside the unit circle provided 
the cycle Tabcd is deformed appropriately. In general case 



^abcd — C'o,! + X! X! (C'a;qfe,e ~ C^- 
x=a,b,c,d k>0 
\x\q''>l 



(A.16) 



e being small enough for Ca,±ig±fc ^ to encircle only one pole of the denominator. 

The following formulas are useful when studying the classical and nonrelativistic 
limits of the quantum Ruijsenaars model. Both correspond to h ^ 0, q = — >■ 1 
and differ only in the behaviour of i ( |3.5| ). As g t 1, 



(x; q). 



x] 



nV^n-l 



{a)k ■= a{a + 1) . . . {a + k - 1), 



ai. 
Pi,. 



1 Pn- 



(A.17) 
(A.18) 

(A.19) 



where n-Pn-i is the standard (generalized) hypergeometric series, and finally (see 
§ 2.5, Corollary 10): 



ln(a;; q)^ = -h ^ Li2(a;) + - ln(l - x) + 0{h), x e (0, 1). 



(A.20) 



Appendix B. Operator Ma(3 



In this section the results are collected concerning the two-parametric general- 
ization Mai3 of one-parametric operator family M = Mg^2g studied in the main text. 
It is an open question whether Map provides a SoV for some integrable model. 

Let us substitute into the Askey-Wilson integral weight w{a, b, c, d; t) ( [A. 141 ) the 
values 

a = sq'2, b = s q'^, c = rq'2, d=r q^ , (B.l) 
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and introduce the notation (quantum analog of ( |2.3U| )) 

Cqi^P] X, y) := {uxy, iyxy'\ ux'^y, ux^^y'^; q)oc- 

The kernel 



l^apii", s \ t) := w{sq2 , s q^^rq^^r q'^;t) 



£5(52 ;s,t) Cq(q2-r, t) 



defines the integral operator 



27ri J t 

prs 

^ Q/3 



the contour F^^ being obtained from Tabcd ( |A.l(j| ) by substitutions ( |B.1D . 
Using the Askey- Wilson integral ( |A.15|) we obtain then the formula 



Kap : 1 



(1 - q){q; g)^ £5(9"^^ r, s 
Now we introduce the operator M^/s 

so that M : 1 — > 1, and having the kernel 

Maf3{r,s I t) 



{l-q){q;q)l{t^,t-^;q)ooCg 


q 2 ;r,sj 


2Bg{a,f3) C,{q'l;s,t) 




q2;r, 





(B.2) 



:b.3) 



(B.4) 



:b.5) 



(B.6) 



:b.7) 



The kernel M. ( ^^) studied in Section 4 is obtained from A4a/3 (|B.7|) after the 
substitutions 



a = 9, (3 = 2g, q- 



r = t+ = y+£2, s = y_, t = t_. 



(B.8) 



It is natural to think of M^is as acting on the space Ref(t) of reflexive (invariant 



w.r.t. t ^ t ^) Laurent polynomials in t. Consider a Laurent polynomial R 
Ref(t), ji,2,ki,2 e Z>o 

Khkik.it) ■= (g^st,gtst-i;g)^.^(gfs-it,gts-ir^g),, 
X {q2rt,q2rt ;q)ki{q^r t,q2r t ;q)k^. 

Using the obvious identity 



'jij2kik2 



(B.9) 



Q fc]^ —k2 Ji — J2 



(B.IO) 
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together with (|B.5D, we obtain the formula for action of Map on Laurent polynomials 



,a/3 (g";g)ii+i2(?'^;?) 



fcl+fe2 



/ £±£ N . iLh£ _1 N 



The set of polynomials Rj^j^kiki ^^^^ enough to choose from it a basis in Ref(t), 
for instance 

:= Ko^oit) = (q^rt, q^rt-\, u = 0,1,2,.... (B.12) 

More correctly, since p^{t) = (^—iyq'^('^^^+l^y'^r'^(^t'^ + t^'^)+lower order terms, 
p^{t) is a basis in the space of reflexive Laurent polynomials in variable t with 
coefficients from Ref(r). The specialization of the formula 

provides thus a tool for calculating explicitely the action of M on any polynomial 
G Ref(t). 

Analysing the formula ( p.l3|) one obtains the following statement. 

Proposition 13 Let f G Ref(t) and suppose f has the parity a that is f{—t) = 
i-iyfit). Let Map: f F. Then: F G Ref(r) ® Ref(s), F{r,s) = F{s,r)\^^p, 
F{-r,-s) = {-iyF{r,s). 

The integral operator Map simplifies drastically when one of the parameters a, 
P takes negative integer values. 

Proposition 14 Let a G Z<o. Then Map turns into the q-difference operator of 
order —a: 

Map:f{t)^Y.Ur,s)f{q'+^s) (B.14) 

k=0 

where 



efc(r,s) = (-l)V 



—a 
k 



-I Q 

{q 2 rs,q 2 r 's;q)k[q ^ rs \q 2 r ^;q).a-k 1 c,^ 

{q''^^;q)-a{q-'s-';q)i-a ' ^ ' ' 



Proof. Instead of analyzing the degeneration of the integral operator defined 
by the kernel (p.?| ) it is easier to study the action of Map on the basic polynomials 

m (EH)- 
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Substituting f{t) := pf^{t) into ( |B7[^ and using ( [B.13| ) we obtain, after simpli- 



fication, the equality 



a + l3 



k=0 



I " + 1^ \ / 1 N 

(g 2 rs;q)k{q 2 rs ^;g)_a_fc 



(B.16) 



which, after substituting (p.l5| ) and making a series of elementary transformations 
(see Appendix I to [l^]), can be put into the form 



k=0 



(l-a-f3-u)k [Q 



2 ^"rs.q 2 r s,q s ,q^ 2 s, —q^ 2 s; q)k 



{q,q 2 +^rs,g 



r ^s,qs'^,q2 s, —q^ s; q) 



a-j3 



identical to the summation formula (11-21) from pO|] : 



(B.17) 



6^5 



a,qa2 , —qa ^ ,b,c, g" aq 
1 1 aq aq !_„ ! ? 



6c 



fag ag \ 



(B.18) 



for the following identification of the parameters 

a 2 7 ^ -1 

a = q s , = q '2 r s, c = q 



(B.19) 



By the symmetry a <-> /5, r <-> s the similar statement can be proved for (3 E'L 



<o- 



To determine the inversion of M^^ let us think of r as a parameter and of M^p as 
an operator M^^ : Ref(t) — > Ref(s) : f{t) F{s). Then, applying M^^ to the basis 
Py{t) ( [B . 1 2| ) and inverting the formula ( |B.13D , we obtain the following statement. 



Proposition 15 The inversion formula for Ml^i^: 
The corresponding kernel is 

(1 - g)(g; g)L(s^ s'^; g)^ Cq(q^]r, t) 



(B.20) 



2£g(-a,a + /3)£q(g 2;s,t)i:g(g"t ; r, s 



(B.21) 
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